Post-occupancy surveys and/or interviews are a common tool used in architecture to evaluate the success of buildings. They can be very useful and should be implemented as long as architects do not expect or claim too much from them. Much has been said of their benefits, but it is concerning to see some architects present them as some kind of scientific proof of a design’s success or failure. Although I am a strong advocate for post-occupancy surveys, I think a little pushback is necessary. A brief review of their methodological weaknesses should make any architect pause before claiming a survey has vindicated their ideas.
First, obtaining a reasonably random sample of people is difficult in any survey. In the case of a single building, like a school, a survey is that much more vulnerable to selection bias. Unless the staff and students were randomly assigned to the school the survey will inevitably be biased. This means the post-occupancy surveys inadvertently bias the results toward certain socioeconomic levels. It will not be representative of the population at large. This makes it difficult to tell where the socioeconomic effects end and the architectural effects begin.
Even if a building is a representative sample of the general population, the users who feel more strongly about the building can influence the survey. They will be more eager to complete the survey and give feedback. These squeaky wheels will unintentionally or intentionally create a selection bias that favors their interests. Those less interested in the subject are less likely to fill out the survey. Subsequently, the survey might only reflect an extreme view that few other occupants agree with.
Therefore, not only is it unreasonable to expect a representative sample from a single building, but it is also difficult for a survey to obtain a truly random sample among the users in a single building. If say 70% percent don’t fill out the survey then a positive or negative response rating of 80% is meaningless. Remarkably and unethically, some do not even list the response rate when they report on the results of a survey. It is important to know how a study’s authors factored in non-responses to their assessment.
These problems with selection biases help fuel the critics who believe architecture is too unique and subjective to be studied. However, if the data are reliable, architects can control for demographic issues by pooling together and cross-referencing multiple studies (meta-analysis). Of course, this is only possible if the original data are reliable. This leads us to the second methodological hazard of post-occupancy surveys. The wording of the questions and the reliability of the responses can play havoc with the data. For example, depending on the sequence of the questions, certain aspects of the design can look more salient to the users. The following hypothetical example demonstrates how this can happen.
How satisfied are you with the design? Do you have natural lighting and a view outside?
The correlation between these two questions could be extremely low. Natural lighting and a view outside might not come to the interviewees’ mind when they are asked to assess their satisfaction with the overall design. This isn’t necessarily true when asked in the reverse order.
Do you have natural lighting and a view outside? How satisfied are you with the design?
Psychologists have shown that reordering questions in this way can dramatically increase the correlation between the two questions. “Satisfaction with the design” is not an easy assessment. It requires factoring in a good number of factors including the cultural, economic, and social impact. The question about lighting and views is a much easier question. By switching the ordering people tend to unconsciously use a heuristic shortcut to answer the more difficult question. Even though the questions are exactly the same, people that are presented the second sequence would most likely be answering how satisfied they are with the views and lighting and not the overall design.
Even more disheartening, well presented questions don’t guarantee the responses will be accurate reflections of the building’s true effects on satisfaction. As a subjective measure, self-reporting isn’t always reliable, and in certain circumstances it is highly unreliable. If a survey is conducted in person the interviewees might be inclined to give less than honest answers to appear kinder and more open-minded to the interviewer. For example, controversial issues such Privately Owned Public Spaces might elicit politically correct answers rather than candid ones.
Social desirability’s effect on surveys, however, is not necessarily my greatest concern when it comes to self-reporting. Social desirability revolves around the idea that people are to some extent “faking it”. The cognitive biases we all struggle against are enough to make self-reporting unreliable without anyone faking it. For example, in the book How Doctors Think, Dr. Jerome Groopman interviewed a surgeon who fervently believes his “outcomes are better than anything in the published literature.” Yet, “when pressed, he admitted that the long-term follow-ups are rare and that he has not participated in any randomized prospective controlled trials comparing fusion surgery with conservative measure such as physical therapy.” Despite the research at the time suggesting fusion surgery only works for a small specific group of patients (those with fractured spines or spinal cancer) Groopman spoke with several other spine surgeons who adamantly refused to enter into a trial comparing simple discectomy with fusion because they are absolutely convinced of fusion’s value. These surgeons have a financial incentive to do more invasive procedures like fusion surgery, but I do not believe they are faking their belief in the procedure, I just doubt the accuracy of their self-reported effectiveness. Imagine how effective fusion surgery would look if we based our conclusion on a survey of these surgeons. Likewise, building users and architects are prone to the same kind of self-reporting cognitive biases that could skew a survey. Architects and users could be absolutely convinced a certain design intervention increased satisfaction when it did nothing of the sort, and perhaps it was even counterproductive.
With only a survey to go on, separating fact from fiction is extremely difficult if not impossible. Post-occupancy interviews and surveys offer a quick and rough way to identify potential factors that increased or decreased the users’ satisfaction. These factors can then be bolstered by other metrics such as absenteeism and productivity and examined more rigorously in observational studies. Surveys on their own are inadequate to draw conclusions about a building. They can play an important role as long as we don’t ask or claim more from them than what they already provide.
If you enjoyed this article check out more by Christopher N. Henry here.
Photographs: albertogp123
Seethaler, Sherry. Lies, Damned Lies, and Science: How to Sort through the Noise around Global Warming, the Latest Health Claims, and Other Scientific Controversies. FT Press. January 23, 2009, Location 2273-2276.
Medical researchers can make a drug or procedure look more effective and harmless by not reporting on those that dropped out of the study. The subjects could have dropped out of study for various reasons unrelated to the procedure or drug, or because of ill effects experienced from the intervention. See: Evans, Imogen, Hazel Thornton, Iain Chalmers, Paul Glasziou and Ben Goldacre. Testing Treatments: Better Research for Better Healthcare. Pinter & Martin; 2nd edition 2011 Kindle Location 1550-80.
This is purely an hypothetical example as there does seem to be a fair degree of correlation of views and natural lighting with users satisfaction. See: Ulrich, Roger. “Biophilic Theory and Research for Healthcare Design.” In Biophilic Design, edited by Stephen R. Kellert, Judith H. Heerwagen, and Martin L. Mador,. Hoboken, New Jersey, John Wiley & Sons, 2008
But this “truism” might not be true in all cases See: Henry, Christopher. “Architecture for Autism: Exterior Views,” ArchDaily.com April 04, 2012.
The questions I used here are hypothetical examples as this phenomenon of substitution has not been studied in architectural surveys. It has been studied in a variety of other surveys, and it seems likely that it would be true of other surveys. For a discussion on substitution see: Kahneman, Daniel. Thinking, Fast and Slow. Farrar, Straus and Giroux, 2011 Kindle Location 1854-1881.
Seethaler, Sherry. Lies, Damned Lies, and Science: How to Sort through the Noise around Global Warming, the Latest Health Claims, and Other Scientific Controversies. FT Press. January 23, 2009, Location 2276-2284.
Groopman, Jerome. How Doctors Think. Houghton Mifflin Harcourt, 2008 Location 3194-3298.
